Spatiotemporal data partitioning for distributed random forest algorithm: Air quality prediction using imbalanced big spatiotemporal data on spark distributed framework

Spatiotemporal air quality datasets are typically collected hourly in monitoring stations deployed non-uniformly across a metropolitan city. These not only big, which poses challenges on the storage and processing capacity of centralized computing systems but also imbalanced spatially heterogeneous, may result biased prediction. To address these challenges, we designed developed parallel prediction system equipped with spatiotemporal data partitioning method, distributed machine learning algorithm, Hadoop’s platform its resource scheduler/manager, Spark’s efficient in-memory execution environment, is suitable for running iterative algorithms, e.g., learning. Our proposed method accounted imbalance spatial heterogeneity features big predictive models, comply load-balancing requirement systems. Distributed Random Forest algorithm H2O library Spark framework was selected as to develop model. This an ensemble forest algorithm-level adjustments perform efficiently possible datasets. An application Tehran, Iran showed that had considerable speedup gain improved both overall accuracy class precision when working A future research direction add streaming visualization functions provide rapid reliable supporting environmental health management.